Skip to content

HDDS-13883. Support HTTPS with ozone insight command#9285

Merged
adoroszlai merged 5 commits intoapache:masterfrom
sarvekshayr:HDDS-13883
Nov 19, 2025
Merged

HDDS-13883. Support HTTPS with ozone insight command#9285
adoroszlai merged 5 commits intoapache:masterfrom
sarvekshayr:HDDS-13883

Conversation

@sarvekshayr
Copy link
Contributor

@sarvekshayr sarvekshayr commented Nov 12, 2025 •
edited
Loading

What changes were proposed in this pull request?

The ozone insight command only supported HTTP connections. This PR adds HTTPS support by checking the ozone.http.policy configuration and selecting the protocol (HTTP/HTTPS) and port (9876/9877 for SCM, 9874/9875 for OM) based on the configured HTTP policy.

What is the link to the Apache JIRA

HDDS-13883

How was this patch tested?

Added a new unit test class TestBaseInsightSubCommand to test the host resolution logic in BaseInsightSubCommand.

Added robot test ozone-insight.robot to run in both secure and unsecure clusters.

Tested manually in ozonesecure docker cluster:

bash-5.1$ ozone insight log scm.node-manager
[SCM] 2025-11-14 06:47:55,575 [DEBUG|org.apache.hadoop.hdds.scm.node.SCMNodeManager|SCMNodeManager] Processing Layout Version report from [datanode=ozonesecure-datanode-2.ozonesecure_default]
[SCM] 2025-11-14 06:47:55,576 [DEBUG|org.apache.hadoop.hdds.scm.node.SCMNodeManager|SCMNodeManager] Processing Command Queue Report from [datanode=ozonesecure-datanode-2.ozonesecure_default]
[SCM] 2025-11-14 06:47:55,781 [DEBUG|org.apache.hadoop.hdds.scm.node.SCMNodeManager|SCMNodeManager] Processing Layout Version report from [datanode=ozonesecure-datanode-1.ozonesecure_default]
[SCM] 2025-11-14 06:47:55,781 [DEBUG|org.apache.hadoop.hdds.scm.node.SCMNodeManager|SCMNodeManager] Processing Command Queue Report from [datanode=ozonesecure-datanode-1.ozonesecure_default]
[SCM] 2025-11-14 06:47:55,796 [DEBUG|org.apache.hadoop.hdds.scm.node.SCMNodeManager|SCMNodeManager] Processing Layout Version report from [datanode=ozonesecure-datanode-3.ozonesecure_default]
[SCM] 2025-11-14 06:47:55,796 [DEBUG|org.apache.hadoop.hdds.scm.node.SCMNodeManager|SCMNodeManager] Processing Command Queue Report from [datanode=ozonesecure-datanode-3.ozonesecure_default]

bash-5.1$ ozone insight log om.key-manager    
[OM] 2025-11-14 06:58:00,016 [DEBUG|org.apache.hadoop.ozone.om.KeyManagerImpl|KeyManagerImpl] Unable to get file status for the key: volume: vol-0-73037, bucket: bucket-0-62485, key: .Trash, with error: No such file exists.
[OM] 2025-11-14 06:58:00,021 [DEBUG|org.apache.hadoop.ozone.om.KeyManagerImpl|KeyManagerImpl] Unable to get file status for the key: volume: vol-0-73037, bucket: bucket-1-87720, key: .Trash, with error: No such file exists.
[OM] 2025-11-14 06:58:00,022 [DEBUG|org.apache.hadoop.ozone.om.KeyManagerImpl|KeyManagerImpl] Unable to get file status for the key: volume: vol-1-22884, bucket: bucket-0-30778, key: .Trash, with error: No such file exists.
[OM] 2025-11-14 06:58:00,023 [DEBUG|org.apache.hadoop.ozone.om.KeyManagerImpl|KeyManagerImpl] Unable to get file status for the key: volume: vol-1-22884, bucket: bucket-1-01823, key: .Trash, with error: No such file exists.

bash-5.1$ ozone insight log om.protocol.client
[OM] 2025-11-14 06:59:33,967 [DEBUG|org.apache.hadoop.ozone.protocolPB.OzoneManagerProtocolServerSideTranslatorPB|OzoneProtocolMessageDispatcher] OzoneProtocol DBUpdates request is received

@sarvekshayr
Copy link
Contributor Author

sarvekshayr commented Nov 12, 2025

@errose28 requesting you to review this PR.

adoroszlai
adoroszlai reviewed Nov 12, 2025
View reviewed changes
Copy link
Contributor

@adoroszlai adoroszlai left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks @sarvekshayr for the patch.

@errose28
Copy link
Contributor

errose28 commented Nov 12, 2025

Can we add integration or acceptance tests to verify that the connection can actually be created after the configs are resolved?

@sarvekshayr
Copy link
Contributor Author

sarvekshayr commented Nov 14, 2025

Can we add integration or acceptance tests to verify that the connection can actually be created after the configs are resolved?

Added a Robot test that runs in both secure and unsecure clusters.

The ozone insight log command in a secure cluster takes time to start printing the output to console. Please let me know if this increased timeout is fine. Else, I can remove the log command test and keep only the metrics one.
I’ve included sample ozone insight log outputs in the PR description tested manually.

@adoroszlai
Copy link
Contributor

adoroszlai commented Nov 17, 2025

The ozone insight log command in a secure cluster takes time to start printing the output to console.

Is that because of the higher heartbeat interval? ozone has custom (5 sec), while ozonesecure uses default (30 sec).

ozone/hadoop-ozone/dist/src/main/compose/ozone/docker-config

Line 48 in 200a243

OZONE-SITE.XML_hdds.heartbeat.interval=5s

@sarvekshayr
Copy link
Contributor Author

sarvekshayr commented Nov 18, 2025

The ozone insight log command in a secure cluster takes time to start printing the output to console.

Is that because of the higher heartbeat interval? ozone has custom (5 sec), while ozonesecure uses default (30 sec).

ozone/hadoop-ozone/dist/src/main/compose/ozone/docker-config

Line 48 in 200a243

OZONE-SITE.XML_hdds.heartbeat.interval=5s

Thanks for the suggestion. After setting the heartbeat interval config to 5s, the logs started appearing much faster. I’ve now reduced the timeout accordingly.

adoroszlai
adoroszlai reviewed Nov 18, 2025
View reviewed changes
@adoroszlai
Copy link
Contributor

adoroszlai commented Nov 18, 2025

@errose28 would you like to take another look?

@adoroszlai adoroszlai merged commit 9ca907c into apache:master Nov 19, 2025
44 checks passed
@adoroszlai
Copy link
Contributor

adoroszlai commented Nov 19, 2025

Thanks @sarvekshayr for the patch, @errose28 for the review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@adoroszlai adoroszlai adoroszlai approved these changes

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sarvekshayr @errose28 @adoroszlai